Markup of a Test Suite with SGML

Recently, there have been various attempts to set up a test suite covering the syntactic phenomena of a natural language (cp. [Flickinger et al. 1989], [Nerbonne et al. 1993]). The latest e ort is the TSNLP project (Test Suite for Natural Language Processing) within the Linguistic Research and Engineering (LRE) framework sponsored by the European Union (cp. [Balkan et al. 1994]). These test suites are meant for the testing of NLP software regarding their coverage of syntactic phenomena. [Volk 1995] showed that a well-organised test suite can also be used to support incremental grammar development and grammar documentation. The key issues in the organisation of a test suite are the ease of extensibility and interchangeability as well as the avoidance of redundancy. We have implemented a test suite, which is optimized for the avoidance of redundancy and we report on the trade-o for extensibility and interchangeability. We de ne a test suite as a collection of syntactically well-formed natural language sentences and contrastively selected ungrammatical sentences (termed: non-sentences). The non-sentences can be used to check for overgeneration of the grammar. The collection is built according to the following principles: